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English documents of EPP (Preliminary 




version) 



1999-03-02 



CAUTION: Some of these documents are not proofed by Ichisugi. There must be a lot of 
mistranslation which are technically incorrect. 



epp-slides 199901 17.pdf(582KB;i 

Slides describing reseach purpose and overview of EPP. 

epp-parser.pdf(233KB) 

Yuuji ICHISUGI: Modular and Extensible Parser Implementation using Mixins(DRAFT) 
(Japanese version of this paper is in Information Processing Society of Japan, Transaction on 
Programming, Vol.39 No.SIG 1(PR0 1), pp.61--69, Descember, 1998. ) 

epp-tvpe-check. pdff 1 98KB) 

Yuuji ICHISUGI: Extensible Type System Framework for a Java Pre-Processor : EPP (DRAFT) 
(Japanese version of this paper v^ill appear in proceedings of SPA'99.) 

EPP documents for plug-in programmers (DRAFT) [edocalpha.zip( 177KB)] 

Detailed documents including API document generated by javadoc. 
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Abstract 

This paper describes a method to construct high- 
ly modular and ext.ensible recursive descent, pars- 
er. This parser is used in an extensible Java pre- 
processor, EPP. EPF can be extended by adding 
plug-ins which extend Java syntax and add new lan- 
guage features. The EPP's parser consists of small 
mixins. A recursive descent parser class is construct- 
ed by composing the^se mixins. The syntax accepted 
by the parser can be extended by adding new mix- 
ins. 

1 Introduction 

The author has developed an extensible pre- 
processor for Java[GJS96], EPP[IR.97, Irii] whicJi can 
be extended by adding new modules whirJi ext.end 
Java syntax, and added new language features. This 
paper describes a method of constructing a high- 
ly modular and ext-ensible recursive descent parser, 
whirJi is the parser of EPP. 

In designing EPP, we aimed at a wide-range of ex- 
tensibility, highly flexible implementation of exten- 
sion modules, and simultaneous usability of multiple 
ext.ension features; - hereafter called composability. 

Traditionally, extension of a system requiras edit- 
ing and direct modification of the source code. This 
has been considered the best method of realizing the 
highest extensibility and fleocibility in implementa- 
tion. With this method, however, it is diflftcult to 
simultaneously use more than one independent ex- 
tension: the user cannot achieve composability. 

Many ext.ensible languages define a new syntax 
and its semantics with a declarative description. 
While this realizes high composabiHty, descriptive- 
ness of a descriptive language limits extensibility and 
flexibility. 

EPP realize>s a.s high ext,ensibility as editing source 
code and realizes as high composability as declara- 



tive de^scription. EPP's parser was implemented in 
sucJi a way as to treat its extension modules as ordi- 
nary general-purpose programming language mod- 
ules, using mixins, the feature of programming by 
diflPerence. The parser has the following features? : 

Plug- ins whicii define new syntax, operators and 
others can be added afterward. Because syntax ex- 
tension plug-ins are programmed by difference, mul- 
tiple plug-ins can be combined at the same time. 

Because of high modularity, plug-ins can be sepa- 
rately compiled one by one. Syntax extension plug- 
ins can be distributed without source codas. 

Because of its general-purpose language descrip- 
tion, it is easy to perform sucJi ad-hoc processes 
as context sensibility and global escape, whirJi are 
rather awkward with the BNF method. 

Error recovery and line number managing can be 
implemented. 

The EPP's arciiitecture is applicable to any other 
programming languagpi? if the implementation lan- 
guage has symbol and mixin features. 

This paper consists of seven more sections. 
Section? outlines EPP, Sections presents symbol- 
s and mixins-language features necessary in pars- 
er implementation, Sections4 and 5 explain how to 
describe the extensible parser using mixins and to 
implement ad-hoc processes, Section6 evaluates the 
parser implementation, Section? describes related s- 
tudies, and Sections is the conclusion. 

2 Outline of Extensible Java 
Pre-processor EPP 

EPP is an extensible Java source-to-source pre- 
processor whicJi can introduce new language fea- 
ture's. The user specify EPP plug-ins at the 
top of the Java source code by writing #epp name 
in order to incorporate various extensions of Java. 
Multiple plug-ins be retrieved simultaneously a.s 
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long as they do not collide with each other. Emit- 
ted source codes can be compiled by ordinary Java 
compilers and debugged by ordinary Java debuggers. 

EPP works not only as an extensible Java pre- 
processor but also as a language-ejcperimenting tool 
for language researchers; a framework for extensible 
Java implementation; and a framework for a Java 
source code parser /translator. 

The EPP's source code is written in Java ext.end- 
ed by EPP itself, and it was bootstrapped by EPP 
written in Common Lisp[Ste90]. The byte-code is 
available in any platform where Java is supported. 



3 symbol and mixins 

This section describees symbols and mixins, whicJi are 
the language features necessary in implementing the 
extensible parser. 

3.1 Symbol Implementation on Java 

Symbols, a data-type as in languages like Lisp, have 
the following feature^s: 

Symbols are similar to constants defined by C lan- 
guage's enum statement. Unlike C constants, howev- 
er, the user can use symbols having arbitrary names 
(strings) as required in the source codes, without 
defining a finite number of elements beforehand. 

Symbols are similar to string literals, but un- 
like strings, the mere pointer comparison efl[iciently 
judges equality of symbols. 

By specifying a name, a symbol can be generated 
dynamically. 

Fig.l shows a program using the Symbol plug-in. 
A Symbol literal is expressed by a colons followed by 
an identifier or a string literal. 

The Symbol plug-in implements symbols on Ja- 
va as follows: The symbol literals in the pro- 
gram are translated into references to each in- 
dividual private static final variable. The vari- 
able is initialized by the return vahie of a stat- 
ic method invocation : Symbol, intern ("name"). 
Symbol. Intern ("name") looks up a hashtable and 
returns an instance of Symbol class having the spec- 
ified name if it already exists. If it does not, a new 
instance is generated, registered in the table and re- 
turned. As a result, symbol literals having the same 
name are guaranteed to reference the instances hav- 
ing the same identity. The execution doas not pro- 
duce the overhead of hashtable retrieval because only 
a static constant value is referenced. 



#epp jp.go. etl.epp.Symbol 
import jp.go. etl. epp.epp .Symbol; 

public class TestSymbol { 

public static void main(String args[]){ 
Symbol X = :foo; 
Symbol y = :"+"; 

System. out .println(x == :foo); // true 
System. out .println(y == :foo); // false 

} 

} 

Figure 1: A program using Symbol plug-in 

3.2 Mixin Implementation on Java 

3.2.1 What is mixins? 

Usually, in an object-oriented language, a particular 
super class name is specified when defining a sub- 
class; a mixin Ls a subclass defined with no particular 
wsuper class specified. A mixin is defined on the pre- 
sumption that multiple inheritance and linearization 
by another class will determine a super class after- 
ward. 

Braclia[BC90] showed that the mixin meciianis- 
m can simulate the same inheritance mechanisms as 
SmallTalk, BETA and CLOS. VanHilst[\^N96] sug- 
gested a variation of mixin-based inheritance whicJi 
enhances reusability of object-oriented programs. 

C-hH- provider? multiple inheritance, but doe^i not 
linearize super classes. Therefore, it is impossible to 
do mixin-based inheritance with the CH-+'s multi- 
ple inheritance mechanism. A class whicli does not 
make a shared super class by multiple inheritance 
is sometimes called a mixin by C+-I- programmers, 
however, it should not be confused with the mixin 
used in this paper. 

A mixin is largely similar to decorator pattern in 
De^sign Pattern [GRV95], with two differenceii: (1) a 
mixin does not allow modules to be exrJianged dur- 
ing execution and (2) a mixin enables a new method 
to be added to a class, while a decorator can only 
have pre-fixed interface. 

3.2.2 Program example using mixin 

EPF's "SystemMixin phig-in" provides mixin-based 
inheritance ^. The following deiicribes the mixin 

^ Unlike intrinsic mixins, SystemMixin plug-ins actually 
provide programming by ciifTerence in the entire system con- 
sists of multiple classpji, not in each class. However, this paper 
regards System Mixins and intrinsic mixins as the same thing 
because RPP's parser consists of only one class. 



class Foo { 

void m(char c){ 
if (c == 'B') { 

doBO; 
} else if (c == 'A') { 

doAO; 
} else { 

doDefaultO ; 

} 

} 

} 

Figure 2: A method definition which uses nested if 
statements. 

features using the syntax e;fct,ended by SystemMixin 
phig-in. First, Fig. 2 shows a common method defi- 
nition using ne^sted if statements. 

Next, Fig.3 shows the method definition divid- 
ed into three mixins. Here, the method- in vocation 
expression, original, introduced by SystemMixin 
phig-in, corresponds to the super method-invocation 
in traditional object-oriented languages. 

Thus, mixins enable the user to divide a method, 
whirJi used to be an inseparable unit, into multiple 
"method fragments" ; and afterward the whole class 
can be constructed by combining multiple mixins. 

3.2.3 Mixin implementation 

EPF's SystemMixin plug-in implements mixins by 
translating all the method fragments incorporated 
in mixins into small Java classes. The method- 
invocation expression searcJies the hashtable of the 
receiver object for the object to implement a method 
segment and is translated into an expression to invo- 
cate its nested call . The problem with this imple- 
mentation is that, with low efficiency, it is impossible 
to treat the mixin- defined class and the intrinsic Ja- 
va class equally. However, this implementation is 
adopted because it allows separate compilation of 
the mixins. 

C+-i- can also implement mixin-based inheritance 
by parameterizing the super class using the template 
mecJianism. Fig.4 shows mixins defined by the tem- 
plate. Combining Skeletons A and B provides the 
class BjAjSkeleton;^;,. Note that mixins defined by 
the template do not allow separate compilation. 

3,3 Mixins Composing EPP 

The EPF's parser is defined a.s the only class named 
Epp, with the class definition divided into multiple 



SystemMixin Skeleton { 
class Foo { 

define void m(char c){ 
doDefaultO; 

} 

} 

} 

SystemMixin A { 
class Foo { 

void m(char c){ 

if (c == 'A') { doAO ; } 
else { original (c) ; } 

} 

} 

} 

SystemMixin B { 
class Foo { 

void m(char c){ 

if (c == 'B') { doBO; } 
else { original (c); } 

} 

} 

> 



Figure 3: A method definition by mixins. 



class Skeleton { 
public: 

void m(char c){ doDefaultO; } 

}; 

template<class Super> 
class A : public Super { 
public: 

void m(char c){ 

if (c == *A*) { doAO ; } 
else "C Super: :m(c); } 

} 

}; 

template< class Super > 

class B : public Super { 

public: 

void m(char c){ 

if (c == 'BO { doBO; } 
else { Super: :m(c); } 

} 

>; 



Figure 4: Mixins defined by the template mecJianism 
of C++. 
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mixins. Starting EPP combines all the mixins com- 
posing the standard Java parser and mixins compos- 
ing the plug-in specified at the top of the source code 
to construct one parser (the class named Epp). EPP 
then generates; the class instance and invokes the s- 
tarting method to begin processing the input source 
code. 

4 Implementation of extensi- 
ble parser 

4.1 Representation of tokens 

One problem about extensible lexical analyzer is how 
the programmer extends the definition of data type 
returned by the lexical analyzer. Two possible solu- 
tions are as follows: 

To provide the means of extending the token data 
type definition along with the means of ext.ending 
the lexical analyzer. 

To provide a general purpose data type in advance 
for all possible tokens. 

Possibility (1) will require modification and re- 
compilation of the source code of the parser because 
the lexical analyzer extension cJianges the data type, 
affecting the parser whirJi processes the data type. 
Therefore, EPP implements possibility (2). The 
EPP's token data types, constructed so as to han- 
dle a wide range of extension, apply to almost any 
language ext,ension without modifying data types. 
More specifically, all the tokens are ejcpre^ssed either 
in literal data types or symbol data types. 

A literal is composed of a tag repre>senting the kind 
of the literal and a string representing the content 
of the literal. For instance, an integer literal 123 is 
expressed as a tag int and a string 123. In this 
way, a literal which was not included in the original 
syntax can be expressed with a new tag assigned. 

All the tokens except literals-identifiers, key words 
such as if and while, operators, and special ciiar- 
acters sucii as semicolons and parentheses-are ex- 
pressed as symbols. EPP does not distinguish be- 
tween keyw^ords and identifiers; therefore, a new syn- 
tax can easily be added by simply extending the 
parser without modifying the lexical analyzer at all. 

4.2 Recursive Descent Parser 

This cJiapter describes the conventional implementa- 
tion of the method of parsing non-terminals without 
using mixins. (Chapter 4.3 describes the implemen- 
tation split into mixins.) 



The following is an example of the production with 
alternatives of a prefix operator, parentheses, a right 
associative binary operator, a left associative binary 
operator, and a postfix operator ^. 

Exp • •++ Exp I ( Exp ) I Term += Exp I Term 
I Exp + Term I Exp ++ 

R^iwriting this production provides the form that 
can be parsed by the recursive descent parser: LL(1) 
grammar. (See the appendix for the details.) The 
recursive descent parser [ASU87] consists of functions 
whicJi parse corresponding non- terminal and return 
the parsed abstract syntax trees. 

Fig. 5 shows a part of a recursive descent parser 
(without mixins) for the non- terminal Exp in the ex- 
ample production, where the three methods return 
abstract syntax trees sls follows: 

• expTop parses alternatives that are neither right 
recursion nor left recursion. For ejcample, a pre- 
fix operator or parentheses. 

• expRight parses alternative of right associative 
operators. 

• expLef t parses alternatives that are left recur- 
sion. For example, postfix operators or left as- 
sociative operators. 

The roles of methods invoked from the program 
are as follows: 

• lookahead returns the token currently being 
noticed. 

• match reports an error if the current token d- 
iffers from the argument value. Otherwise, it 
discards the current token and reads the next 
token. 

• matchAny discards the current token uncondi- 
tionally. 

The program generates the abstract syntax trees 
as is expected from the production. For example, a 
-\-= b += c generates (+— a (-f = be)); a -h b + c 
generates (+ (-h a b) c). 

4.3 Extensible Recursive Descent 
Parser 



^Tn fact, in general language grammar, a non-terminal nev- 
er mingles alternatives of parentheses and a binary operator; 
or a right associative binary operator and a left asaociative 
binary operator. Such grammar cannot generate expressions 
like a + (b) or a + b += c, and then, the grammar conflicts 
with human intuition. 
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Tree exp() { 

Tree tree = expTopO ; 
while (true){ 

Tree newTree = expLef t (tree) ; 

if (newTree ~ null) break; 

tree = newTree; 

} 

return tree; 

} 

Tree expTopO { 

if (lookaheadO == :"++"){ 
matchAny 0 ; 

return new Tree( : "prelnc" , expO) 
} else if (lookaheadO == :"("){ 
matchAny 0 ; 
Tree e = exp() ; 
match( :")"); 

return new Tree( : "paren" , e) ; 
} else { 

return expRight (expl() ) ; 

> 

} 

Tree expRight (Tree tree) { 
if (lookaheadO == :"+="){ 
matchAny 0 ; 

return new Tree (:"+=", tree, expO ) 
} else { 

return tree; 

} 

} 

Tree expLeft(Tree tree) { 
if (lookaheadO == :"+"){ 
matchAny 0 ; 

return new Tree (:*'+", tree , expl() ) 
} else if (lookaheadO == :"++"){ 
matchAny 0 ; 

return new Tree( : "postinc" , tree) 
} else { 

return null; 

} 

} 

Tree expl() { return term() ; } 



SystemMixin Exp { 
class Epp { 

def ine Tree exp () ■£ 
Tree tree = expTopO; 
while (true){ 

Tree newTree = expLef t(tree) ; 
if (newTree == null) break; 
tree = newTree; 

} 

return tree; 

} 

define Tree expTop(){ 

return expRight (expl 0 ) ; } 
define Tree expRight (Tree tree){ 

return tree; } 
define Tree expLef t (Tree tree){ 

return null; } 
define Tree expl(){ 

return term() ; } 

} 

} 

Figure 6: A skeleton of a ejctensible parser. 



SystemMixin Plus { 
class Epp { 

Tree expLef t (Tree tree) { 
if (lookaheadO == :"+") { 
matchAny 0 ; 

return new Tree (:"+*', tree , expl ()) ; 
} else { 

return original (tree) ; 

} 

} 

} 

} 



Figure 5: A part of a recursive deijcent parser. 



Figure 7: A mixin which defines a left associative 
binary operator. 
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Splitting the program shown in Fig. 5 into mixins 
makes it more modular and extensible. Removing 
if-then clauseii and leaving eAse clauses in Fig.5 pro- 
vides a skeleton as shown in Fig.6. Exp defined by 
the mixin is a method of parsing the following pro- 
duction: 

Exp • •Term 

New alternatives can be added to non-terminals 
by extending the methods expTop, ejcpRight and ex- 
pLeft in this program using mixins. Fig. 7 shows a 
mixin which defines a left associative binary opera- 
tor. 

EPF defines dozens of kinds of non-terminals as a 
set of the mixins whicJi define the skeleton as shown 
in Fig.6 and the mixins whirJi add alternatives; as 
shown in Fig.7. With, a macro faciUtating these def- 
initions, the mixins in Fig.6 can be defined by the 
following one line: 

def ineNonTerininal(exp, termO); 

Also, the mixins whicJi add the left associative bi- 
nary operator in Fig. 7 can be defined by the fol- 
lowing one line: 

def ineBinaryOperator (Plus , : , exp) ; 

4.4 Lexical Analyzer Extension by D- 
ifFerence 

A recursive descent lexical analyzer is ext.ensible by 
difference. The EPF's lexical analyzer mainly con- 
sists of the following methods, whose behavior can 
be extended by mixins, 

readToken 

read Id 

readNumber 

readOperator 

readStringLiteral 

readCharLiteral 

readTraditionalComment 

readEndOf LineConment 

For example, Fig. 8 shows the mixin that doe^s not 
regard // as a beginning of a comment if it is fol- 
lowed by : . (This feature preserves extensibility and 
compatibility with Java. This is an example of a pro- 
gram using the mixin. 

//: assert (predicate) ; 

This line is simply regarded as a comment by the s- 
tandard Java compiler, but works as an assert macro 
if the file is processed by EPF. ) 



SystemMixin CommentPragma -i 
class Epp { 

Token readEndOf Line Comment 

(EppInputStream in){ 
if (in.peekcO == 
in.getcO ; 

return readToken (in) ; 
} else { 

return original ( in) ; 

} 

} 

} 

} 

Figure 8: A mixin whicJi extends the lejcical analyz- 
er. 

4.5 Parser Module Deletion and Re- 
definition 

EPP also provides a means of extending grammar 
other than programming by difference. 

Grammar extension by programming by difference 
is somewhat limited in that (1) only new parser mod- 
ule addition is possible; current module deletion is 
impossible and (2) ejctension works only with pre- 
pared "hooks" (methods). 

One way to perform extension without program- 
ming by difference is not to execute the original 
method invocation in adding mixins: i.e., to disre- 
gard the original method. For example, redefining 
expl in Fig.6 modifies the precedence of operators. 

In addition to this, plug-ins have a mean of re- 
moving some parser definition modules (mixin) when 
plug-ins are loaded. By the mean, the grammar can 
be arbitrary modified. 

The drawback of realizing the plug-ins through the 
above two means is that they are much le^ss compos- 
able than extension only by programming by differ- 
ence. A plug-in programmer /user has to trade-off 
extensibility against composability. 

5 Ad-hoc Processes 

5.1 Backtrack 

EPP provides explicit badctrack with the lexical an- 
alyzer and the parser. Fig. 9 shows a mixin whicii 
defines a new token "**" . 

The argument EppInputStream is an input stream 
that bacJctracJcs at an arbitrary length by having the 
whole input file as a character array on the memory. 
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SystemMixin NewOp { 
class Epp { 

Token readOperator(EppInputStream in){ 
if (in.peekcO == '♦*) { 
int p = in.pointerO ; 
in.getcO; 

if (in.getcO == { 
return :"**"; 

} 

in . backtrack (p) ; 
return original (in) ; 
} else { 

return original (in) ; 

} 

> 

} 

} 

Figure 9: A mixin which defines a new token. 

5.2 Context Sensitivity 

A recursive descent parser easily implements 
context-sensitive processes in the lexical analyzer 
and the parser: the user Just has to input the contex- 
t information into the global variables (static vari- 
ables, in Java terminology). 

5.3 Error Recovery 

Error recovery during parsing is easily implement- 
ed by Java's exception handling feature. The error 
handler just has to skip tokens till a particular token 
appears. 

5.4 Line Number Managing 

It is desirable to have information on "the line num- 
ber at which the syntax began" in the Tree that is 
generated by parsing. The information helps gen- 
erate clear error me>ssages when errors occur during 
semantic analysis afterwards. EPP also uses the in- 
formation to output eacJi line of the source code at 
the same line after translation. 

Lisp and Java easily implement line number man- 
aging. Lisp uses variables with dynamic scope, and 
Java uses static variables, starJcs and try-finally syn- 
tax. 

Fig. 10 shows an example definition of the method 
exp with Java. All the non- terminal methods are 
defined in the same way. The Tree constructor ob- 
tains the line number at which it began by riierJcing 



Tree exp()-C 

LineNumber . stack 

. push ( current LineNumberO ) ; 
try { 

. . . Same as Fig. 6 . . . 
} finally { 

LineNumber . stack . pop ( ) ; 

} 

} 

Figure 10: Managing Une number information. 

the top of the starJc of the static variable LineNum- 
ber. starJc. 

6 Evaluation 

6.1 Java Grammar Description 

EPP incorporatas a complete Java parser for JDK 1.1 
implemented as described in this paper. The Java 
grammar definition part consists of 105 mixins, 29 of 
whirli define the skeleton of non-terminals as shown 
in Fig.6. 

Explicit bacJctracic was executed at the following 
points during implementation: 

1. Distinction between constructor and method or 
field. 

2. Distinction between static method/field and 
static initializer. 

3. Distinction between method and field. 

4. Distinction between local variable declaration 
and statement. 

Some of the above can be parsed with LL(1) by 
rewriting grammar, but that lowers ext.ensibility and 
modularity. Therefore, hacktracJc was adopted. 

Field access syntax and cast syntax are imple- 
mented with lower modularity; it is impossible to 
make these elements highly modular unlp>ss type in- 
formation is obtained during parsing. 

6.2 Efficiency 

The author tested the speed of EPP for proceiising 
of source code of EPP itself, consisting of 7218 lines, 
and obtained the following results. 

• MMX Pentium 233MHz, Windows95, Microsoft 
SDK2.01: approximately 30 seconds 
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• UltraSPARC 200MHz, Solaris2.5.1, JDK1.L3: 
approximately 40 seconds 

The process consists of three parts: source code 
parsing, macro expansion of ext.ended syntax, and 
source code emission after translation. The most 
time-consuming part is parsing. That is no problem 
for practical use, but a parser should work much 
faster. Three factors cause EPF's low speed: 

1. Java interpreter overhead, 

2. Mixin method invocation overhead, and 

3. intrinsic low speed of parser description meth- 
ods proposed by this paper (e.g., sequential 
searcii of alternatives with if- then-else and un- 
nece^ssary invocation of method having no con- 
tent).' 

The overhead of the current mixin method invo- 
cation is approximately 10 timas slower than that of 
the standard Java class invocation. This is because 
EPP implements a mixin-defined fragment with a s- 
mall Java object, invoking the method by searcMng 
the hashtable during execution. Mixin implementa- 
tion speed should be improved in the future. 

The overhead of sequentially searriiing for alterna- 
tives with if- then-else takes up mucJi time. In order 
to improve that, one possible sohition is to imple- 
ment an optimized translator that is specialized to 
the EPP's parser source code. For e>xample, if-then- 
else should be translated into a table searcii. 

In general, having too many bacJctracks reduces 
parsing efficiency, but Java grammar never caus- 
es bacic tracks that will seriously reduce the speed, 
as proven by the following experiment. When 
the source code of Java.uil. Vector in JDK 1.1.1 was 
parsed by EPF, 1214 tokens were appeared. There 
were 86 backtracks whirJi caused 172 times ext.ra in- 
vocation of readTokens method. The result shows 
that the invocation of readToken increases approxi- 
mately 14 

6.3 Ease of Debugging 

Standard declarative description parser generators 
detect, collisions and ambiguity of grammar to warn 
the user. Unfort.unately, EPF's parser doe^ not have 
this feature. Since it is intended for ext.ending the 
grammar of already completed languages by differ- 
ence, the parser is not intended as a tool for design- 
ing a new language grammar. 

Nevertheless, it would be possible to construct a 
parser generator whicJi generates mixins described 
in this paper. 



Loc^ debugging of a recursive descent parser Ls 
easy; the standard Java debugger and print state- 
ments work as in common programs. 

7 Related work 

ANTLR,[ANT] and JavaCCfJav] are recent top-down 
parser generators based on LL(k). According to 
the creators of theiie tools, the advantages of a top- 
down parser include ease of debugging and passing 
attribute valuers downward or upward during pars- 
ing. A bottom-up parser like LALR(l) does not have 
sucii feature^!. Also, ANTLR can extend the existing 
grammar by diflFerence through inheritance. JavaC- 
C enables direct writing in part of the production, 
making writing easy with declarative de>scription. 

MP C + + [Ish94] , Open C -f- + [Chi9 5] , 

JTRANS[KK97] and OpenJava[Tat] are ext.ensible 
systems whirJi can introduce new language fea- 
ture's by providing compile time MOP (Meta object 
Frotocol[KdR,B91]) during compilation. Like EF- 
P, their task is to perform complicated translation 
on an abstract syntax tree after parsing. Also, the 
grammar is extensible in a limited range. For ejcam- 
ple, MPC+-}- allows addition of new operators and 
statements. 

Eli[GHL"'"92] is a compiler-generator whicJi mod- j 
ularizes the grammar definition. It automatically 
generate^s a language processor using grammar and 
semanticii definitions based on attribute grammar, 
and defines a new language by a kind of inheritance 
using existing definition modules. 

Many "extensible languages" for grammar modifi- 
cation have been created, and most of them, includ- 
ing Lisp and C macros, define new grammar exten- 
sion in the on-the-fly style, i.e. in the program to 
be proceijsed. The problems with on-the-fiy ext.en- 
sion are that (1) it does not allow pre- compilation of 
the grammar extension code and therefore lacks effi- 
ciency, and (2) modifying or extending "the syntax 
to define syntax extension" itself often causes confu- 
sion. EPP has no sucii problems because it does not 
work in the on-the-fiy style. 

Camlp4[RAu] Ls an Objective Campl pre-proce>ssor 
whose grammar can be extended by adding modules. 
The extension can be done by difference with declar- 
ative description, and the modules can be compiled 
separately. 

8 Conclusion 

A method of constructing a highly modular and 
extensible parser by splitting the recursive descent 
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parser into small mixins was described. The syntax 
accepted by the parser can be e;>ctended with high 
compos ability over a wide range by adding mixin- 
s implemented by programming by difference. Al- 
so, in principle, removing existing mixins arbitrarily 
modifies grammar. 
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A Production Rewrite 

The production defined as follow is rewritten so that 
it can be parsed by the recursive de^scent parser. 

Exp • •++ Exp I ( Exp ) I Term += Exp | Term 
I Exp + Term I Exp ++ 

Split the production into two by introducing 
ExpTop. 

Exp • 'ExpTop I Exp + Term I Exp ++ 

ExpTop • •++ Exp I ( Exp ) I Term += Exp | Term 

Riimove the leit recursion of Exp by introduc- 
ing ExpLoop and rewrite ExpTop by introducing 
ExpRight. 
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Exp • 'ExpTop ExpLoop 

ExpTop • •++ Exp I ( Exp ) I Term ExpRight 
ExpRight • •+= Exp I • • 

ExpLoop • • + Term ExpLoop | ++ ExpLoop I • • 

Now this grammar can be parsed by recursive de- 
scent parser. Furthermore, by introducing ExpLef t, 
ExpLoop can be rewritten as follows: 

Exp • 'ExpTop ExpLoop 

ExpTop • •++ Exp I ( Exp ) I Term ExpRight 
ExpRight • •+= Exp I • • 
ExpLoop • • ExpLef t ExpLoop | • • 
ExpLef t • •+ Term I ++ 

Fig. 5 shows a part of a recursive descent parser for 
non- terminals ba^sed on the above productions. Note 
that the tail recursion of ExpLoop is rewritten into 
Loop and embedded in the method exp; the method 
expLeft expresses returns a special value, null, if no 
m atching alternatives . 

Both the original and rewritten grammars are am- 
biguous. For example, the expression ++ a + 1 can 
be interpreted as either (++ (+ a 1)) or (+(++ a 
) 1) . The program in Fig. 5 parsers this as (+(++ a 
)1)- 
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EPP 1.1.0 Architecture Overview 



Required Knowledge 

In order to understand EPP documents, you will need basic knowledge of the Java lanaguage as well as 
basic knowledge of compilers. Specifically, you must understand the following words. 

pass 

recursive descent parser 

token 

literal 

AST (Abstract Syntax Tree) 

non-terminal 

bootstrap 

However, advanced expert knowledge should not be required. 

Furthermore, the following concepts familiar to lisp programmers appear. 

macro, immutable object, symbol, S-expression, backquote macro, dynamic variable 

I shall explain these concepts as much as I can, so that readers who are not experienced in lisp can 
understand them. 

EPP Description Language, Ld-2 

The source code of EPP is written using Java language that is extended with EPP. Specifically, the 
following five plug-ins are used. 

Symbol 
SystemMixin 
BackOuote 
AutoSplitFiles 
EppMacros 

The source code for these five plug-ins are also written using these five plug-ins. Other than this, we 
have the Common Lisp source code for the EPP main portion, and source codes for five plug-ins. Using 
these to bootstrap, EPP can work even if you only have Java. 

Of the five plug-ins, the SystemMixin plug-in is especially important. The SystemMixin plug-in 
implements a new object oriented language called Ld-2 on top of Java. 



(Application Programs) 
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EPP Plug-ins 

EPP Core / Java Grammar definition 
Object-Oriented Language Ld-2 

JavaVM 



The Ld-2 lanaguage has a special inheritance mechanism that is different from that of the Java language. 
With this mechanism, a single class can be divided into multiple "components" called "mixins" v^hich 
can be described separately. A class is built up by merging the multiple mixins. 




In the current implementation, Ld-2 classes created by merging mixins are not compatible with classes 
of the Java language, and are defined using a different syntax. The syntax for calling methods is also 
different. 



EPP Main Routine 

When EPP is invoked, a program named EPP main routine executes. The program is a Java class with 
the following name. 

j p , go . etl . epp . epp . Epp 

The EPP main routine builds an EPP preprocessor with different configurations for each file to be 
processed, and invokes those preprocessors. 

An EPP preprocessor is an Ld-2 class created by merging multiple mixins. A preprocessor for a specific 
file is built by creating an Ld-2 class combining the mixin for the plug-in that was specified at the 
beginning of the file, and the "mixin that defines the standard preprocessor". 

Plug-ins can only extend the behavior of EPP preprocessors. They cannot extend the behavior of the 
EPP main routine. However, by creating a subclass of the jp.go.etl.epp.epp.Epp Java class, you can 
create a customized EPP main routine. 

EPP executes the translation process on a file-by-file basis by default. However, if you specify the - 
global option, EPP goes into Global Processing Mode and will process all files globally. 



http://staff.aist.go.jp/y-ichisugi/epp/edoc/edocalpha/outline.html 
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(D mixin 



Java :SCii^ 
mixin 





JWllVCTWNlWtPtl-lM^^WMHWMWMHHMWMMfJVllWW^ ^ . ^ , ^^^^ ^ ^ ^ ■ . . . . . _ _ j _ _ 

EPP Preprocessor 

When the EPP preprocessor is invoked from the EPP main routine, the initialization method is called. 
After that, the input file is processed through the following four passes: parsing pass , macro expanssion 
pass , type checking pass , and code emitting pass . 

The parsing pass will call the lexical analyzer as required. The lexical analyzer will read character-by- 
character from EppInputStream . a class that is the input stream for the EPP. 

Plug-ins can extend the EppInputStream, lexical analyzer, parsing pass, macro expanssion pass, type 
checking pass, and code emitting pass. 

You can also add additional passes prior to, and after the four passes. For further information on adding 
passes, refer to EPP Preprocessor Core . 



TcJoen 



char 



Paisei 



Tree I 



Lexical analysr 



EppIi^iSineam 



Tiee 



Type chedcing 



TlBG 



Code endttiz^ 
char I 



Data Structure 



Within the preprocessor, three data structures that describe the token, abstract syntax tree , and typ e are 
particularly important. 
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Tokens are data types that are returned by the lexical analyzer. 

The abstract syntax tree is created by the parsing pass, and then translated by the type checking pass, 
passed to the code emitting pass for conversion to character strings and finally written to the output file. 

The abstract syntax tree has nodes that have type information and those that do not. The type checking 
pass converts an abstract syntax tree without type information into an abstract syntax tree with type 
information. 

The three data types mentioned above are all immutable objects. That is, you cannot modify their 
internal state from within a program. 



Plug-ins cannot define subclasses of a class that describes tokens or abstract syntax trees. The data 
structure of tokens and abstract syntax trees are very versatile and new tokens and syntaxes can be 
expressed without adding new subclasses. 



The Principle of the Extendable Parser 

The parser is basically written in recursive descend style. You can add a mixin and extend the method of 
the parser in order to add a new alternative to the non-terminal. By backtracking and proceeding with 
context sensitive processing, the parser can handle non LL(1) type syntaxes. It can also handle left 
recursive rules. 

For further information please refer to the following, paper: 
" □ ■ , if^r^f. ■ . f%ofSfQfB ,JES^'£0 « .8Z □ . Aa\>Foo8 □ ISi " 



Related Information 

For further information regarding writing plug-ins, please refer to the following. 
Error Handling 
Dynamic Variables 



The source code for the EPP main portion is located under epp/src/levelO/epp of the distributio package. 
The major fuctions of the files are shown below. 



The Role of Each Source File 



Epp . java 
EppCore . java 
EppInputStream. java 
Lex . j ava 
CompUnit . j ava 
TypeDecl . java 
Statement . j ava 
ExpNonTerm. java 



EPP main routine definition 
EPP preprocessor main portion 
EPP input stream 
lexical analyzer 

Java program top level syntax definition 

class, interface, method, field syntax defintion 

statement syntax definition 

non-terminal definition related to expressions 



http://staflf.aist.go.jp/y-ichisugi/epp/edoc/edocalpha/outline.html 
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Exp.java expression syntax definition 

TypeSystem. java definition of the Type class and definition of type semantic 

TypeCheck. java definition of the standard Java type checking object 

FileSig.java definition of class types and seperate compilation processin 



Index 
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Web Images Groups News Frooqle Local more » 

E^yA f^eprocessp^^ 1999 I^M^S^ 



Web Results 1-10 of about 113,000 for JAVA preprocessor 1999. (0.23 seconds) 

SOS : Java Preprocessor!!!! 

SOS : Java Preprocessor!!!! dtv48@canal-plus.fr dtv48@canal-plus.fr Thu, 18 Nov 
1999 18:30:42 +0100. Previous nnessage: SOS : Java Preprocessor! ... 

https://lists.xcf.berkeiey.edu/ !ists/advanced-java/1 999- November/003582, html - 5k-- Csched - Similar pages 

SOS : Java Preprocessor!!!! 

SOS : Java Preprocessor!!!! Yogesh Pandeya yogesh@indts.com Thu, 18 Nov 1999 
22:31:39 -0000. Previous message: Rmiregistry; Next message: SOS : Java ... 

htt.ps://lists.xcfberke!ey,edu/ lists/advanced-java/ISSS'-November.'OOSSBO.htmj - 4k - Jun 8, 2005 - 

CacTied - SjmijaLpaaes 

EPP: An Extensible Pre-Processor Kit 

1999-11-05 (EPP 1.1.0 beta 10) Error recovery in type-checking pass was implemented. 
... "Extensible Java Preprocessor Kit and Tiny Data-Parallel Java" ... 

staff.aist.gojp/y-ichisugi/epp/ - 10k - Cached - SjMlf^r.PfMI^s?. 

Comp.compllers: Looking for a cpp(c preprocessor) written in Java 

Looking for a cpp(c preprocessor) written in Java bharath3@my-dejanews.com (1999-05-03) 
... I am looking for an implementation of c preprocessor in java. ... 

compilers.iecc.com/compafch/articie/99-^15-013 - 3k - Cached - Simliar pactes 

Visustin - C/C++. C#, Java. JSP. JavaScript and PHP flow charts 

... support is built according to the ECMAScript standard (ECMA-262 3rd Edition 
1999). ... C/C++, C#, Java: Preprocessor directives are treated as comments. ... 

wvAv.aivosto.com/visustin/help/c -java. html -- 8k - Cached - ShYijlar paries 

Java News from December, 1999 

Monday, December 27, 1999. Sun's released version 1.2 of the Java ... ut wasn't 
the preprocessor one of the C obfuscations Java was supposed to save us from ... 

www.cafeaulait.org/1999december.htmi -25k - Cached vSjMlfPT.ir^ages 

The Jonathan IDL2Java compiler 

The compiler automatically launches jpp a simple preprocessor which implements a 
... It maps individual IDL module scopes to JAVA packages and overrides the ... 

jonathan.objectv^eb.org/current''doa^jrers.'^dl2Java.html - 27k - Cached - Similar pagea 

Jfront - Operator Overloading for Java 

The Java Grande working group has been working on this problem too. ... however we 
would rather see this preprocessor become a part of a mainstream ... 

www.winternet.com/-gginc/jfront/ - 6k - Cached - Sjjtilar.pa^jes 

The Jonathan IDL2Java compiler 

The Jonathan idl2java Compiler generates the java classes and interfaces necessary 
... The compiler automatically launches jpp a simple preprocessor which ... 

pauinacJnria.fr/cdrom/ www/jonathan/doc/hrefs/idi2Java.html -- 27k Cached. - SimjlLir.gage 
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JavaML: A Markup Language for Java Source, bv Grec J. Badros 
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In this paper, I introduce the Java Markup Language, JavaML — an XML application for 
... A framework for preprocessor-aware C source code analyses. ... 

www9.org./w9cdronn/342/342.html - 80k - Cached - §ifr|ilaLR4!af4i^. 
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December 1999 Java News 

Friday, December 31, 1999 

Hot on the heels of version 2.2, Slava Pestov has released version 2.3prel of JEdit the popular 
pure Java programmer's text editor, version 2.3 adds predefined abbreviation expansion, more 
toolbar buttcons, and assorted user interface improvements and bug fixes. JEdit is released under 
the the GNU General Public License. 



Macintosh Runtime for Java 2.2 EA2 expu-es at midnight tonight. This isn't really a Y2K failure, 
just abunch of programmers underestimating how much time would be required to complete the 
next version. Starting tomorrow users will periodically see a dialog asking them to download a 
new version of the software.If you ignore the dialog, MRJ will still function correctly. 



Steev Coco's released version 1 . 1 of Seymour , an open source Java IDE written in pure Java but 
mostly tested on the Macintosh. 



I ^ ^ ^P IBM's released the second beta of the free Aglets Software Development Kit LI 



Aglets are mobile agents written in Java. JDK 1 . 1 is required. JDK 1 .2 will not work. 



Sun's posted an early access release of the Personal Java Runtime Environment for Windows CE 
2.11/MIPS Platfonn on the Java Developer Connection (registration required). It allows you to 
run PersonalJava L 1 . 1 applets and applications on devices running Windows CE 2. 1 1 on a MIPS 
processor. 

Thursday, December 30, 1999 

Sun's posted a proposal for archiving/serializing Swing and JavaBeans based GUIs as XML . This 
may help fix the problem of inconsistent binary serialization formats between JDK versions that's 
especially troublesome for Swing applications. The proposal includes a sample Bean Builder tool 
for building GUI code from the proposed XML grammar. JDK 1.3 is required. 

Wednesday, December 29, 1999 

My essay on Free Art? Free Software? got picked up on Linux Today and other sites and mailing 
Usts; and consequently I'm getting a lot of feedback, even in the relatively dead time between 
Christmas and New Year. I'm on vacation right now myself, so I can't respond to all of it 
immediately. I should get to it next week (barring the collapse of civilization as we know it). One 
of my New Year's resolutions is to get discussion forums working on this site; but in the 
meantime, Linux Today is hosting discussion of the article. 
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Monday, December 27, 1999 

Sun's released version 1 .2 of the Java Embedded Server, a small footprint application server for 
devices like cash registers and photocopiers. This is payware, but a 90-day free trial is available. 
Version 1 ,2 adds support for Java 2, Secure Sockets Layer (SSL), and the latest version of Swing, 
among other features. 



Sun's released version 1 .0.2 of the Java Advanced Imaging API for Windows and Sparc Solaris. 



Version 1 . 1 .3 of the Java 3D API OpenGL based implementation for Sparc Solaris and Windows. 



Sun's posted version 0.95 of the JavaPhone Specification and Documentation in Acrobat PDF 
format. (At the end of the millenium, don't you think it's about time Sun's engineers and spec 
writers learned to use HTML like everybody else? Scott McNealy allegedly banned PowerPoint 
from Sun a couple of years ago. Perhaps it's time he banned Word and FrameMaker as well.) 
There's also class library documentation in the Javadoc HTML format. 

Sunday, December 26, 1999 

I've published an essay about some problems I see with Richard Stallman's call for free 
documentation . I'd be interested in hearing people's reactions and thoughts. Mostly I codified 
these thoughts while listening to RMS talk at The Bazaar a couple of weeks ago, and then reading 
a couple of his own documentation efforts for emacs and gdb. 

Friday, December 24, 1999 

Hot on the heels of version 2.2, Slava Pestov has released version 2.2. 1 of JEdit the popular pure 
Java programmer's text editor, to fix three bugs. JEdit is released under the the GNU General 
Public License. 

Wednesday, December 22, 1999 

I'm away on vacation in New Orleans for the holidays. Updates are liable to a be a little sporadic 
here until the New Year. 



The Apache Jakarta Project has released Tomcat 3.0 . the official reference implementation of the 
Java Servlet API and Java Server Pages. 

Monday, December 20, 1999 

Version 2.2 final of Slava Pestov's JEdit pure Java programmer's text editor is now available. 
JEdit is published under the the GNU General Public License. 
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IBM's alphaWorks has released some Enterprise JavaBeans for its WebSphere Application Server 
including Company Components, Address Components, Currency Components, and Natural 
Calendar Components. 

Saturday, December 18, 1999 

Sun's posted release candidate 2 of the JDK 1.2.2 for Linux on the Java Developer Connection 
(registration required). This release fixes assorted bugs. 



Sun's posted final releases of four specifications: 

• Java Server Pages 1 . 1 

• Java Servlet API 2.2 

• Enterprise JavaBeans I.I 

• Java 2 Enterprise Edition 1.2 

Sun's released version 1.2 of the Java Embedded Server , a small footprint server for use in 
vending machines, pay phones, copiers, and so forth. It's not clear what the price is. It probably 
requires a contract with and royalty to Sun for actually shipping a product. However, as 90-day 
evaluation version is available for free. 

Sun's released the Sun BluePrints Design Guidelines for J2EE. an integrated set of documentation 
and examples that illustrates "best practices" for developing and deploying J2EE compatible 
solutions. These are supposed to give developers of e-commerce applications examples of 
component design and optimization, division of development labor, and allocation of technology 
resources. 

Friday, December 17, 1999 

Neil Taylor's released the first beta of version 1 .2 of Jake, a visual front end to javac, javap, and 
other Sun command line tools. 



Beta 4-1 of Colt 1.0 is now available. Colt The Colt distribution provides an open source 
infrastructure for scalable scientific and technical computing in Java that contains data structures 
and algorithms for offline and online data analysis, linear algebra, multi-dimensional arrays, 
statistics, histogramming, Monte Carlo simulation, parallel and concurrent programming. 
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Version 0.1.81 of Tritonus . a GPL'd implementation of the JavaSound API for X86 Linux, is now 
available. This release can write .au files, adds clips for esd, and fixes assorted bugs. 

Thursday, December 16, 1999 

Sun's released the first beta of version 2.0 of the server-optimized HotSpot virtual machine for 
Windows . This beta requires JDK 1.2.2. Sun claims a 30% performance improvement over 
version 1.0. 



Sun's also posted version 1.0 of the Java Transaction Service (JTS) specification . According to 
Sun, 

JTS specifies the implementation of a Transaction Manager which supports the 
JavaTM Transaction API (JTA) 1.0 Specification at the high-level and implements 
the Java mapping of the OMG Object Transaction Service (OTS) 1.1 Specification at 
the low-level. JTS uses the standard CORBA ORB/TS interfaces and Internet Inter- 
ORB Protocol (HOP) for transaction context propagation between JTS Transaction 
Managers. 

A JTS Transaction Manager provides transaction services to the parties involved in 
distributed transactions: the application server, the resource manager, the standalone 
transactional application, and the Communication Resource Manager (CRM). 

Tuesday, December 14, 1999 

Sun's posted the first public review draft of version 1 .2 of the PersonalJava specification . . 
Comments are due by January 7, 2000. 

Sun's also released a beta of Java Access Bridge for the Microsoft Windows 1.0 . This product 
allows Windows based Assistive Technology to get at and interact with the Java Accessibility 
API. This release adds an installer, fixes some bugs, and adds a couple of things to the API. The 
installer only really works under Windows NT. 



IBM's alphaWorks has released a new version of the Bean Scripting Framework that adds support 
for VBScript/JScript/PerlScript on Win32 platforms. The Bean Scripting Framework is an 
architecture for incorporating scripting into Java applications and applets. 

Monday, December 13, 1999 

Romain Guy's released version 2.6 of his Jext programmer's editor with many new features and 
bug fixes. Jext is written in pure Java. 
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Saturday, December 11, 1999 



IBM's alphaWorks has released the first iteration of cPreProcessorstreanu an inputstream for 
people writing parsers that need C/C++ preprocessing; that is, #inciude, #if , #def ine etc, 
Corbet me if I'm wrong, ut wasn't the preprocessor one of the C obfuscations Java was supposed 
to save us from? About the only plausible use I can imagine for this would be if you were writing 
a C or C++ compiler in Java, and why would you want to do that? 

Friday, December 10, 1999 

I'm afraid I missed most of the Java Business Expo due to a brief bout with the flu. By the time I 
was feeling better, all I had to time to do was see the Penn and Teller show (lots of fun) and make 
a brief pass through the show floor. The floor was noticeable for the large number of booths filled 
with engineers talking to customers about real products as opposed to flashy Ught shows and 
booth bunnies. I really wish I had more time to chat with more vendors. 

The one thing I did had time to look at that looked worthy of further investigation was JVision , a 
$99 payware tool for reverse engineering Java code into UML diagrams. The Linux version is 
free, and all version are free for academic use. I've needed something like this for my books and 
courses for a while now, but everything I've seen up till now cost four figures or more (a price 
point where I just rule the product out of consideration.) It's available on Linux, NT, and Solaris. 
There's a 30-day demo available. I'll try it out and let you know what I think. 



•••'••'■•■■■■■•••■"■"'''••'■"'■^■"■"•"■"•"'•"'•"'■''•■■•■■'•■'•■•'■"■'■•'■•A'A'^.-A-^."^.^^ 



IBM's alphaWorks has posted a new copy of their IRC Client for Java just to extend the expiration 
date to June 30, 2000. (W ouldn't it be easier just to release one that doesn't expire?) 

Thursday, December 9, 1999 

Borland's released JBuilder Foundation for Linux, Solaris, and Windows. This is a free beer, pure 
Java IDE based on Java 2. 

Wednesday, December 8, 1 999 

Sun has officially withdrawn Java from the ECMA standardization process because the ECMA 
refused to be used as a rubber stamp for whatever Sun chose to submit and insisted on retaining 
control of the standard. Earlier, Sun pulled out of the ISO process for the same reason. Sun is 
adamant in its refusal to allow anyone but Sun to have any authority or control over Java. They 
are willing to listen to companies or individuals who have good ideas (as long as they aren't 
Microsoft or Bill Gates, in which case Sun won't even listen) but it simply refuses to put itself in a 
position where anybody else can make it put something in or take something out of Java against 
Sun's expressed wishes. This effectively ends any hope of a Sun-supported, de jure Java standard. 
The ECMA is debating whether or not to go forward without Sun's participation. 

One of the main purposes of de jure standardization like that performe 
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SOS : Java Preprocessor!!!! 

dtv48@canal-plus*fr dtv48@canal-plus.fr 
Thu, 18 Nov 1999 18:30:42 ^0100 



Previous message: SOS : Java Preprocessor! ! ! ! 

Next message: misaligned Imagelcons when seen over http 

Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] 



>i%i'vvvv^ir»yvvvvvvvvvvvsi'V^^ 



try Kiev at this adress 

http : / /www. f orestro . com/ kiev/ home . html 



SeB 

Message d*origine 

De : yoqeshQindts . com [mailto : yoqeshQindts . com ] 

Envoye : jeudi 18 novembre 1999 23:32 

A : Michael . HeistQuni lever . com 

Cc : advanced- java@xcf . berkeley.edu 

Objet : RE: SOS : Java Preprocessor!!!! 



Yeah I need it written in java so that i can bundle it and do the processing 
at install time. . . . 
Do you know any??/ 
Thanks 



Original Message 

From: Michael Heist [SMTP : Michael . Hei stQuni lever . com ] 

Sent: Thursday, November 18, 1999 7:56 PM 

To : voqesh@indts . com 

Cc : advanced-javaGxcf .berkeley. edu 

Subject: Re: SOS : Java Preprocessor! ! ! ! 

does the utility have to be written in java? most c compilers have a flag 
for preprocess only 

so you could precompile your java files with your c compiler, for some 
reason this appeals to 
me greatly. 



#undefine GOSLING 



>Hi List, 

>I need It urgently! ! 

>I need a utility In Java which does pre processing like C or C-h-f-, I want to 
give #ifdef and 

>#eise #endlf In my java code Inside the class definition and preprocess It 

to generate new j ava 

>^iie appropriately. 

>thanks f 

>Yogesh 
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To unsubscribe, mail advanced-j ava-unsubscribe@xcf . berkeley . edu 
To get help, mail advanced- j ava-help@xcf . berkeley . edu 



To unsubscribe, mail advanced-java-unsubscribegxcf . berkeley. edu 
To get help, mail advanced-iava-help@xcf . berkeley . edu 



To unsubscribe, mail advanced-java-unsubscribeGxcf . berkeley . edu 
To get help, mail advanced-java-helpQxcf . berkeley. edu 



• Previous message: SOS : Java Preprocessor! ! ! ! 

• Next message: misaligned Imagelcons when seen over http 

• Messages sorted by: [ date ] [ thread ] [ subj ect ] [ author ] 
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SOS : Java Preprocessor!!!! 



Frank D. Greco f greco@crossroadstech.com 
Thu, 18 Nov 1999 09:31:05 -0500 

• Previous message: SOS : Java Preprocessor! ! ! ! 

• Next message: More offscreen lEditorPane rendering... 

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] 

At 07:03 PM 11/18/99 +0000, Yogesh Pandeya wrote: 
>Hi List, 

>I need it urgently! ! 

>I need a utility in Java which does pre processing like C or C++, I want 
>to give #ifdef and #else #endif in my java code inside the class 
>definition and preprocess it to generate new java file appropriately, 
>thanks , 
> Yogesh 

errr... how about using the C/C++ preprocessor itself? The C/C++ 
preprocessor doesn't know anything about C/C++... its just a 
regular text preprocessor. Or how about even using *m4'? Its 
been around for over 20 years... 

Frank G. 

I Crossroads Technologies Inc, 55 Broad Street, 28th Fl , NYC, NY 10004 \ 
I Email: fgrecoQCrossroadsTech, com Web: www, Cross roads Tech , com \ 

I Voice: 212-482-5280 k229 Fax: 212-482-5281 \ 

+======================================================================+ 



To unsubscribe, mail advanced-j ava-unsubscribe@xcf . berkeley . edu 
To get help, mail advanced-j ava-help@xcf . berkeley . edu 



• Previous message: SOS : Java Preprocessor! ! ! ! 

• Next message: More offscreen JEditorPane rendering... 

• Messages sorted by: [ date ] [ thread ] [ subj ect ] [ author ] 
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Products : feedback 



Java definitions: .0-9 A B C D E F G H I J K L M N O P Q R S I U V W X Y Z 

t 

You are here : home : Java Glossary : P words : preprocessor. 

preprocessor 

Java does not have an official preprocessor. You can alway use C/C++*s, cpp. However, 
Java does guarantee to eUminate dead code inside an if like this: 



static final boolean DEBUGGING = false; 

if ( DEBUGGING ) 
{ 

System , out . pri ntl n (i ) ; 

} 



The code inside the if still has to be valid. Unfortunately you cant use the technique, 
for example, to select one of two different import statements. 



PBC.. PP. n EPP n jpp n jpp n Scrjptic 





horne 
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Dowri^oad Now 



Fast Newsgroups 
Extractor 
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I Home 



Java Preprocessor 0,1 prerelease 



Pubiisher : GehtSoft 



Java Preprocessor allows you to use the C-like 
macros in your Java sources. The preprocessor 
is the superstructure over the Java compiler and 
requires the JDK. The preprocessor translates 
your sources with the preprocessor macros into 
the Java sources and calls the Java compiler. 
The messages about errors are translated into 
Microsoft C-like messages. The positions of 
errors will point to the positions in original files, 
not to the resulting Java sources. 



Download NOW 
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Preprocessor 



antlr.preprocessor 



Class Preprocessor 



java , lang . Obj ect 

I i'i p. \. \ C. . V ii .V i- .V 



L 



P. t .1 r; * 'C . I J k c • a r .r* tfj r 



L 



an tlr .preprocessor . Preprocessor 



All Implemented Interfaces: 

PreprocessorTokenTypes 



public class Preprocessor 
extends LikParser 

implements P[:eprocessprToM 



Field Summary 



static java . lang. String 

[] 



static i.VltiJc'.t 



static ::;if:$!ev 



static B.it:-:ei 



static 1?1 1 1 



static tiir.Se:; 



static B .i t S e t 



static t ;> t. 



tokenSet. 0 



tokenSet. 3 



tokenSet. 6 



http ://www. antlr. org/j avadoc/antlr/preprocessor/Preprocessor. html 



6/10/05 



Preprocessor 



Page 2 of 8 







Static i3.it ;>et 


tokenStsfc 7 




static Bit Set 


cokenSet 8 





Fields inherited from interface 

antlr.preprocessor.PreprocessorTokenTypes 


ACTION, ALT, ARG ACTION, ASSIGN RHS, BANG, CrIAK 


LT^ 




COI^IA, COMMENT, CURLY BLOCK SCARF, DIGIT, ELEMEN 


t7 




HEAD^rlR ACTION, ID, ID OR KEY"V/ORD, LITERAL catch. 






LITERAL cla;^.s, LITERAL except ). c;a, LITERAL exten.ci 






LITERAL private, LITER/vL protected, LITERAL oubi 


ic 


f 


LITERAL retu,>:ns, LITER/>.L throws, LITER/^j tokens, 






ML CC-MI^iENT, NEWLINE, -nTULL TREE LOOKAHE."Xs, OPTION 


s 


STAI<T, 


RCURLY, RPAR^ilN, RliLli; BLOCK, SEMI, SL COMMENT, 






STRING LITERAL, SUBRULE BLOCK, TOKENS SPEC, WS, 







Constructor Summary 

E^.^'P^:<><^^3.^.9.?^:. iS^^IJ^^i^^^J^^:!)!^ state) 



( § ~ . tokenBuf) 



???^^P.^PP.?r.r?.^.^^.^. ( X9ii?i^§liL5iLtr^ lexer) 



Method Summary 


antlr .preprocessor '. Grammar 


class def ( j ava . lang . String file, 
H i r a r c h y h i e r ) 


j ava . lang . String 


exc.eptionGroup ( ) 


j ava . lang .String 


tSiXceptlonHandlar ( ) 


j ava . lang . String 


ejgceptionSpec ( ) 


void 


qr aiEunar Fi le ( H i e r a r c h y h i e r , 
java. lang. String file) 


T v: 6 & a e c L o /. 


( antlr . preprocessor . Grammar gr) 


void 


r epo r tE r r o r ( R e c; c) c] i t i o r ■ E c e d t. i c:- r* e ) 
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Delegates the error message to the 
tool if any was registered via #initTooi 

(antlr.Tool) 


void 


reportError ( j ava . lang . String s ) 

Delegates the error message to the 
tool if any was registered via #initTooi 

(antlr.Tool) 


void 


r^portWaxrniiiq ( j ava .lang. String s) 

Delegates the warning message to 

thp tnnl if ^inv xa/a^ rpni^tprpH iii ni t-foni 

LI Lwwi II OI ly vvao i^uioiwiowj vici trxxix l. i (jljj. 

(antlr.Tool) 


void 


riale (antlr . preprocessor . Grammar gr) 


void 


setTool (Tool tool ) 

In order to make it so existing 
subclasses don't break, we won't require 
that the antlr Tool instance be passed as a 
constructor element. 


j ava . lang . String 


superclass ( ) 


j ava . lang . String 


throws Spec ( ) 



Methods inherited from class antlr.LLkP.arser 

'.^ '.J" J t f-i vJ>.\^l.-- , .".11 , f f V- .1. v." A t A / V- ... '7? Ki^'.-.i.j' '-A V- 



Methods inherited from class antlr.Parser 


c;dc>ies sage Listener, addParserListener , 








a dci P a r e r Ma t c h 1j i t e a e r , a dci Pa r ^i e r T C5 k e n i s t e m e 








a cic is e ma r • t c: P e d i c: a t e X: :i . t e :c i e , ad 6 S y ii t a tic;? r e 


d.i.C'-'ii 


eLi 


stener. 


addTraceXjis tener , coriSUiiieUntil , ccnsumeUntil , 


getA 


1 




aetASTFactor y , qet H'ilename , qe t Input St ate , ae 


tToke 


nNa 




q e t T o k e nN arae s , q e t T o k e nT yp e T o AS T C las sMa.p , i s D 


ebu^ 


ode 


, mark. 


match, match, ruatchNot, panic, rernoveM.e;?;3aqeL 


is ten 


• f 




r evno v €i P a, r s e r L i s t e. n e r , r e. ti\o v e P a r- e ;E:Ma t c: h. lA. ste.ri 


er. 






jcreino ve Parse r T o ke nLi. s t: e ne r , r erao ve S einant: i. c ? red 


icate 




tei^er , 


re?AoveS_^rit acticPredicateLi st.ener , removeTrace 


Liste 


ner 


f 


r e v;i nd , s e t AS T b"'a c t o r v , s e t ASTN ode CI a s s , set AS 


TNode 




- ^ 


e *: V:* eb u aM.o de , s e t F i 1 e. n ame , s e. 1 1 a n o r e I n v a 1 i <;.D e 
- - ' . ,.. ' -I — 


buaCa 






e T ::idu t S ta t e , s e e:To kenBu f. f e r , t r.a c:e I riderj t: 

















Methods inherited from class java.lang.Object 

equals, getClass, hashCode, notify, notifyAll, toString, 
wait, wait, wait 
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Field Detail 



tokenNames 



public static final j a va . lang . String [ ] _tokenMames 



tokenSet 0 



public static final PA tS<ii':. _to]cenSet__0 



tokenSet 1 



public static final riitSet tokenSet 1 



tokenSet 2 



public static final Bl irl^et: tokenSet 2 



tokenSet 3 



public static final Bit:5;^:: _tokenSet_3 



tokenSet 4 



public static final 3itS>j!t: __tokenSet_4 



tokenSet 5 



public static final j?it,;5v^t tokenSet 5 



tokenSet 6 



public static final BitjJeV. tokenSet 6 
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_tokenSet_7 

public static final BitLf^jt token Set 7 



_tokenSet_8 

public static final iLLtiP^dV tokenSet 8 



Constructor Detail 



Preprocessor 

public Preprocessor ( ToX^jt^yi^JiiL^I. tokenBuf) 



Preprocessor 

public Preprocessor ( TGk^^!;\3t; i>-i.-j;r: lexer) 



Preprocessor 

pub 1 i c Pr ^r oce s sor ( P 5 j:.^ S S h^i^^^ state) 



Method Detail 



setTool 



publ i c void setTool { ,Too i tool) 

In order to make it so existing subclasses don't break, we won't require that 
the antlr.Tool instance be passed as a constructor element. Instead, the 
antlr.Tool instance should register itself via #initTooi(antir.Tooi) 

Throws; 

java . lang . IllegalStateException - if a tool has already been registered 

Since: 

2.7.2 



reportError 

public void reportError ( j ava . lang .String s) 

Delegates the error message to the tool if any was registered via #initTooi 
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(antlr -Tool) 



Overrides: 

reoor/uKrroj: in ClaSS Pora«r 

Since: 

2.7.2 



reportError 

public void J^eportError { Kecc.Kx/ri.t^ e) 



Delegates the error message to the tool if any was registered via #initTooi 

(antlr .Tool) 



Overrides: 

^i^^sisJj^^i^jiii in class Pjiyji<iy^_ 

Since: 

2.7.2 



reportWarning 

public void reportWarning (java . lang . String s) 

Delegates the warning message to the tool if any was registered via 

#i nit Tool (antlr . Tool ) 

Overrides: 

in class i-f.^^rsc^r 

Since: 

2.7.2 



grammarFile 



public final void qranimarFile ( iil :^j:;.\r.^:'iv hier, 

java. lang . String file) 
throws •^f^coqi^.i j.f;.tivxof;pr.Aori , 



Throws: 



To k.f? r; t j: e aiv..;:'. x c e r j •: 'i . o ; • 



optionSpec 

public final Irid'^xv^^Vi^'rjuor cptionSpec (antlr .preprocessor .Grammar gr) 

throws K;^coori-i.tAor;Excepr::i or? , 
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Throws: 



R^c 0;^n X t i o ri :^ j^x; ^J?:t ^ o n 
*ro ko> n S t r otar{\E Xi::<?n f; i o n 



class def 



public final antlr . preprocessor . Grammar class_def {java. lang . String file, 

r.xdif.-i'co'r-v hier) 
throws Re-:joori;l •■•..i onn-xcKpL.i on , 

Throws: 



superclass 



public final java.lang. String superclass ( } 

t h r ows i:-^':?.?:tir:P.-.t-}: i-.'^iri, ' 

To k/? n .'i; t ji* arr-.]:) x c r> *: .1 o ri 



Throws: 

T o k -3 n 3 1 ,•: > j -3 rr-.E x c -3 o t i o r 



rule 



public final void rule (antlr .preprocessor . Grammar gr) 

t h r o ws i^iSn - i^i^Vi^^: ^iivA-^- JilH. ' 

To k.e r: 1'. r et'.nOr'.x r> ':. oi •-. 



Throws: 

T :1' k G- ii t .ITG S.jT.i'jXC G t". j_ " 



throwsSpec 



public final java . lang . String throwsSpec ( ) 

t h r ows R<i o. o a \ v. i, o ;i r^xo *i *o t x q , 
To-jcofj S t j; eauv^i >i c e , o ; \ 



Throws: 

To k.e .t: t r e^^iv.Excer^ 1 o > 



exceptlonGroup 

public final java.lang . String exceptlonOroup ( ) 
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throws ReoognA tioTiKxofts^tj.oi:: ^ 
To i'le S t r eaivE >* ctv;; o;v 



Throws: 



exceptions pec 



public final j ava . lang . String except i onSpec ( ) 

throws -^S:ily:3jiiJ:A2})-^-??::^^^^^ 

T'jk&riStre amr; x c o i:; t j. n 



Throws: 

T) o o o n i 1*. i o Vj '/,. o '^i sl"? \. i o 



\ 



exceptionHandler 



public final j ava. lang . String exceptionHandler ( ) * 

throws l^;:vi?c-V;^nA.i;^:i^vU:V^^:?^;:^^^ 



Throws: 

^-j ^ T-^ -j H \ v-', , 'i " <n "i"* * T-^ 

To k'j; n S t i: e.rfi i]'. x c d ■: i o i' < 
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Home I Download | News | About ANTLR | Fe 




,%X."XjA555s% j,j jAs^ 




Latest version is 2.7.5. 
Download nowl » 



. i 



>^ Home 
» Downioad 
» News 

»Using ANTLR 

» Dociimentati on 
» FAQ 

Ailjcjes 
» Grammars 

» Tech.SiiRPP.rt 
»About ANTLR 

» Whatis ANTLR 
» Whv use ANTLR 
» Showcase 
» Testjmoniais 
» Getting Started 
n Software License 
» ANTLR WebLogs 
n ANTLR Workshops 

^^ StringTe mpjate 

*^IML 

» PCCTS 

Feedback 
^Credits 
»Contact 



— ^- — ^- — ^- — — . — . 



ANTLR Reference Manual 



ANTLR 

Reference Manual 



Credits 

Project Lead and Supreme Dictator 
University of San Franciso 

Support from 
Your View of the Java Universe 

Help with initial coding 
John Lilly, Empathy Software 

C++ code generator by 

E§tecWe[ls_ and Ric_Kjarerx 

C# code generation by 
Micheal Jordan, Kunle Odutola and Anthony Oguntimehin. 

Eylhon's universe has been extended by 

WPSl§JMh^M!nS§I and Afer^^.K*;?/^ 

Infrastructure support from Perforce : 
The world's best source code control system 



Substantial intellectual effort donated by 

Jim Coker 

John Mitchell 
Qhwn)3JifJM.Qk (UNICODE, streams) 



Source changes for Eclipse and NetBeans by 

Marco van MeeQen and Brian Smith 



http://www.antlr.org/doc/index.html 



6/10/05 



ANTLR Reference Manual 



Page 2 of 6 



ANTLR Version Z7.5 

January 28, 2005 

What's ANTLR 

ANTLR, ANother Tool for Language Recognition, (formerly PCCTS) is a language 
tool that provides a framework for constructing recognizers, compilers, and 
translators from grammatical descriptions containing Java, C++, or C# actions [You 
can use PCCTS 1.xx to generate C-based parsers]. 

Computer language translation has become a common task. While compilers and 
tools for traditional computer languages (such as C or Java) are still being built, 
their number is dwarfed by the thousands of mini-languages for which recognizers 
and translators are being developed. Programmers construct translators for 
database formats, graphical data files (e.g., PostScript, AutoCAD), text processing 
files (e.g., HTML, SGML), ANTLR is designed to handle all of your translation 
tasks. 

Terence Pa^^ has been working on ANTLR since 1 989 and, together with his 
colleagues, has made a number of fundamental contributions to parsing theory and 
language tool construction, leading to the resurgence of LL(k)-based recognition 
tools. 

Here is a chronological history and credit list for ANTLR/PCCTS. 

See ANJLR.sojtware 

Check out Getting started for a list of tutorials and get your questions answered at 

th e ANTLR. f"AQ.at.igmL- CO m 

See also http://www.ANTLR„ora and aiossary . 

If you are looking for the previous main version (PCCTS 1.33) of ANTLR rather than the new Java-based 
version, see G^MOiL?Mt^<lydthP^^^ 



Download ANTLR. 

ANTLR.2 J,5iejease 
ANTLR Meta-Language 

.1^^..., 

• Header Section 

• .Lex!caJ.Anaiyze 

• I.ree::fiars8r Class 

• Tokens Section 

• Grammar inheritance 

• Rule Definjtions 

• Ai^^n)io Produ(^g 

• -Simple Prpduction.e 
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• TpkejijCiasses 

• EBNF RiLle Eiemjints 

• interpretation Of Semantic Actions 

• %l)iCLCticp[:edicate^^ 

o Rxed depth.ipo 

LB)oc^„Anaiysis 

• Lexical Ruies 

• Common prefixes 

• Tp/K.en.Mributes 

• Scanning Binary Files 

• Scanning Unicocie Characters 

• Manipuiating Token Text and Objects 

• filtering.input.Stre 

o AjJiR^Mii^guei^n^ 
o Ni)ngieedy_SybxuLes 
o Greedy Subrules 

o U(DjtMPJ)s.ofMQna!:^.^dy„^^ 

• LexjcaLStates^ 

• Tfl§...^:nd Of Fije C^^ 

• Case sensitivity 

• jgnonng whitespace in the lexer 

• Tracking Line information 

ANJLR.Tree.Parse^^^^ 

• Wliat's,a^tre_ei>aj^e£f^^ 

• What kinds of trees can be parsed? 

• T!;GeL.g!:QXn!I]aLru!OS 

o SX0taQM??„e!!§dJra 

o SejTianli.c predjc^^^ 

o An Example Tree Waiker 
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Token Streams 

• introduction 

• PcLSSrlhrougJi T 

• T^»hjer].Streani F 

• Toi^n.Stn8am Spitting 

o Example 

o Filter implementation 

o How To Use This Filier 

o Tree Construclion 

o Garbage. CMi^MPiLl 
o Notes 

• jQMn.StreamMlAlt^ 

o Parsing Muiiiplexed Token Streams 

• TokenStreajTiR 

Jol5Bn„ygcabjjJarie$ 

• introduction 

o hlP>v does AN 

o Why do token types start at 4? 

o What flies associated with vocabiilarv does ANTLR generate? 

o How does ANTLR synchronize the symboMype niappings between 
3XanM?M^.M}..!t:e.san3e.Jiie 

• GT.<5.nimarinhe^^^^ 

• Recognizer Generat ion Order 

• Tficicv Vocabuiary Stud 



Error Handling and Recovery 

• AMTLR.ExceptiOiiHijBj;^ 

• Modifyjn5..Defau!tEno 

• Parser Exception Handling 

• Sp.ec[^nj5_P_arse.r Excepfe 

• Default Exception Handitnq in the Lexer 

.Java.Runt!me.MQdel 

• programjiner'sj.rfte^ 

o What ANTLR generates 

• Muitip!e Lexers/Parsers With Shared input State 
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o Pi^rser CJass 

o EBNFjSiJlxiijes 

o ElPdLlQfipjiPjiididbn. 

o Prociucilon Eiernent RecognitiGn 

o Slandard .Cfass^^^ 

o Lexer. Form 

o Creatina Your Own Lexer 
................. ...... . ................................ 

o Lexical Rules 

• Token Objects 

• Token Lookahead Buffer 

.C±±.Runtimg.m.odel 

• 3E^ldlngihej:untjo)e. 

• Channincj the AST Type 

• UsSncj HeterocjeneoLis AST types 

,C#.Runtj.me.niodei 

• Building the ANTLR C# Runtime 

• SpedfvinQ Code General^on 

• C#-Spediic ANTLR Options 

• .A„Temi?lats.C# A 

.Py.thon„R:jntime.m.Q 

• Building the ANTLR Python Runtime 

• Specifying Code Generation 

• Python-Specifio ANTLR Options 

• A Jem^Late Pyjhor^A 

ANTLR. Tree.Coiistruc^iQn 

• Notation 

• Conlroliin p AST construction 

• Grammar annotations for buiidinq ASTs 

o .Lea[.{]o.d.e3 

o Rp-^t.[LQdes 

o Tu.rni.ns.P.ff.slanilard.tr^^ 

o Treji j}Oiie_co.nsjTiLQti^^ 

o ASTj^j^tlPJlJl^nsja^^^^ 

• invoking parsers that buiid trees 

• AST Factories 

• il§M!:P3.erLeous.ASTs 
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o An Expression Tree Example 

• ASTenumejiations 

• A few_ex_ami>lfis 

• kabeLeisjjbjijjes 

• Reference nodes 

• Required AST functionality and form 

.GrafrLniar.lnherita^^ 

• !r!!:r.Q.^y?ti.on.a.nq m 

• FunotEonaitty 

• Where Are Those Supercirammars? 

• Error Messages 

Options 

• nie:.Grammaiian 

o language: Setting the generated iangiuage 
o k: Setting the Sookahead depth 
o ifI!&oftVpcab:JnEl|a^ 
o exgpjtVocab; Na^^^^^ 
o t^?stUleraJs;„ G ene^^^^^ 

o defaujtErr orHandier: Controlling default exception -handiing 
o codeGenM akeSvWtchThresho td: controiiing code generation 
o codeGenBilsetTeslThreshoid: conlroliing code generation 
o buildAST: Automatic AST construction 
o A^j'LabelT^iXJiS 
o cih.arVocabufar^ 
o wainWhenFpJio^ 

• fiomjTianclj^me.0 
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